AFRL-HE-AZ -TP-2000-0007 


UNITED  STATES  AIR  FORCE 
RESEARCH  LABORATORY 


COMPUTER-BASED  INTELLIGENT  TUTORING 
SYSTEMS:  A  COGNITIVE  APPROACH  TO  TEAM 
TRAINING  AND  PERFORMANCE  RESEARCH 


J.  Wesley  Regian 

Air  Force  Research  Laboratory 
Warfighter  Training  Research  Division 
Information  Training  Systems  Branch 
7909  Lindbergh  Drive 
Brooks  Air  Force  Base,  TX  78235-5352 


Linda  R.  Elliott 

Veridian  Engineering 
2504  Gillingham  Drive,  Suite  201 
Brooks  Air  Force  Base,  TX  78235 


August  2001 


AIR  FORCE  RESEARCH  LABORATORY 
HUMAN  EFFECTIVENESS  DIRECTORATE 
WARFIGHTER  TRAINING  RESEARCH  DIVISION 
6030  South  Kent  Street 
Mesa  AZ  85212-6061 


Approved  for  public  release;  distribution  is  unlimited. 


NOTICES 


Publication  of  this  report  does  not  constitute  approval  or  disapproval  of  the  ideas 
or  findings.  It  is  published  in  the  interest  of  STINFO  exchange. 

Using  Government  drawings,  specifications,  or  other  data  included  in  this 
document  for  any  other  purpose  other  than  Government-related  procurement  does  not  in 
any  way  obligate  the  US  Government.  The  fact  that  the  Government  formulated  or 
supplied  the  drawings,  specifications,  or  other  data,  does  not  license  the  holder  or  any 
other  person  or  corporation,  or  convey  any  rights  or  permission  to  manufacture,  use,  or 
sell  any  patented  invention  that  may  relate  to  them. 

The  Office  of  Public  Affairs  has  reviewed  this  report,  and  is  releasable  to  the 
National  Technical  Information  Service,  where  it  will  be  available  to  the  general  public, 
including  foreign  nationals. 

This  report  has  been  reviewed  and  is  approved  for  publication. 


J.  WESLEY  REGIAN 
Project  Scientist 


DEE  H.  ANDREWS 
Technical  Director 


JERALD  L.  STRAW,  Colonel,  USAF 
Chief,  Warfighter  Training  Research  Division 


Please  do  not  request  copies  of  this  paper  from  the  Air  Force  Research  Laboratory.  Additional  copies  may  be  purchased 
from: 

National  Technical  Information  Service 
5285  Port  Royal  Road 
Springfield,  Virginia  22161 


Federal  Government  agencies  and  contractors  registered  with  the  Defense  Technical  Information  Center  should  direct 
requests  for  copies  of  this  report  to: 

Defense  Technical  Information  Center 
8725  John  J.  Kingman  Road,  Suite  0944 
Ft.  Belvoir,  Virginia  22060-6218 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE  3.  DATES  COVERED  (From  -  To) 

August  2001  Final  May  1995  -  December  1998 


4.  TITLE  5a.  CONTRACT  NUMBER 

Computer-Based  Intelligent  Tutoring  Systems:  A  Cognitive  Approach  to  Team  Training  and  F41624-95-C-5007 
Performance  Research  5b.  GRANT  NUMBER 


6.  AUTHOR(S) 

Regian,  J.  Wesley 
Elliott,  Linda  R. 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratory 
Fluman  Effectiveness  Directorate 
Warfighter  Training  Research  Division 
Information  Training  Systems  Branch 
2504  Gillingham  Drive,  Building  125 
Brooks  Air  Force  Base,  TX  78235-5352 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratory 
Human  Effectiveness  Directorate 
Warfighter  Training  Research  Division 
6030  South  Kent  Street 
Mesa  AZ  85212-6061 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


5c.  PROGRAM  ELEMENT  NUMBER 

61102F 


5d.  PROJECT  NUMBER 

2313 


5e.  TASK  NUMBER 

HT 


5f.  WORK  UNIT  NUMBER 

20 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

AFRL 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

AFRL-HE-AZ-TP-2000-0007 


Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

Air  Force  Research  Laboratory  Technical  Monitor:  Dr.  J.  Wesley  Regian,  (210)  536-6812;  DSN  240-6812 


14.  ABSTRACT 

This  paper  describes  progress  related  to  a  program  of  research  in  computer-based  team  training.  We  describe  an  approach  to  development 
of  computer-based  team  training  systems  based  on  principles  of  cognitive  skill  acquisition  and  a  functional  conceptualization  of  teamwork. 
We  consider  individualized  training  technology  and  theory  regarding  individual  skill  acquisition  and  apply  this  approach  to  teamwork 
functions  and  team  decision  processes.  First,  we  review  principles  of  computer-based  individual  training  previously  established  and 
validated  in  our  lab.  We  then  describe  a  functional  taxonomy  of  teamwork  demands,  based  on  core  aspects  of  teamwork  with  emphasis  on 
dimensions  relevant  to  training  issues.  Finally,  we  relate  these  teamwork  dimensions  to  a  taxonomy  of  cognitive  task  demands. 


15.  SUBJECT  TERMS 

Automated  Instruction;  CAI;  Cognitive  Engineering;  Computer-Based  Training;  Intelligent  Tutoring  Systems;  ITS;  Skill  Acquisition;  Team 
Performance;  Team  Training 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

OF  ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF  RESPONSIBLE  PERSON 

Dr.  J.  Wesley  Regian 

a.  REPORT 

UNCLASSIFIED 

b. ABSTRACT 

UNCLASSIFIED 

c.  THIS  PAGE 

UNCLASSIFIED 

UL 

30 

19b.  TELEPHONE  NUMBER  (include  area 
code) 

(210)  536-6812  DSN  240-6812 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39.18 


CONTENTS 


Page 

INTRODUCTION .  1 

Characteristics  of  CAI  Instruction .  2 

ICAI  Conponents .  3 

ICAI  and  Group  Instruction .  3 

ICAI  and  Teamwork  Instruction .  5 

Teamwork  Task  Demands:  A  Functional  Approach .  9 

Determining  Cognitive  Demands:  Cognitive  Taxonomy  of  Teamwork .  1 1 

DISCUSSION .  13 

REFERENCES .  16 

FIGURES 

Figure  1.  Instructional  Engineering  for  Individually  Performed  Tasks .  6 

Figure  2.  Instructional  Engineering  for  Teamwork  Tasks .  7 

Figure  3.  Instructional  Engineering:  Teamwork  and  Taskwork .  8 

TABLES 

Table  1.  Levels  of  Individualization  in  Computer-Assisted  Training  Systems .  2 

Table  2.  Components  of  Intelligent  Computer-Assisted  Training  Systems .  4 

Table  3.  Cognitive-Based  Taxonomy  of  Teamwork .  12 


iii 


COMPUTER-BASED  INTELLIGENT  COMPUTER  SYSTEMS: 

A  COGNITIVE  APPROACH  TO  TEAM  TRAINING  AND  PERFORMANCE  RESEARCH 


INTRODUCTION 

Advancements  in  computer-based  technology  and  quantitative  modeling  tools  represent 
opportunities  to  expand  intelligent  tutoring  and  performance  engineering  technology  to  more  complex, 
ambiguous,  and  dynamic  team-based  settings.  Computer-based  systems  can  now  portray  core 
characteristics  of  complex  multi-operator  task  environments  and  provide  enhanced  platforms  for  training 
and  research  in  naturalistic  settings.  For  example,  military  command  and  control  (C2)  scenarios  have 
been  constructed  to  emulate  the  cognitive  task  demands  of  operational  C2  team  members  using  PC-based 
representations  of  the  task  environment  (Chiara  &  Stoyen,  1997;  Coovert  et  al.,  2000;  Hess  &  Elliott, 
2000;  Hess,  MacMillan,  Elliott,  &  Schiflett,  1999;  Schiflett  &  Elliott,  2000)  which  were  based  on 
extensive  accumulation  of  information  gained  from  approaches  such  as  cognitive  task  analysis,  training 
requirements,  and  focal  group  information  (Elliott  et  al.,  1998,  1999,  In  review;  Fahey,  Rowe,  Dunlap,  & 
deBoom,  1997;  Klinger  et  al.,  1993;  MacMillan  et  al.,  1998). 

Advanced  quantitative  modeling  techniques  and  intelligent  agent  capabilities  allow 
representation  of  other  computer-based  entities,  such  as  hostile  forces  and  fellow  team  members. 
Existing  methods  of  knowledge  elicitation  must  be  adapted  to  more  easily  capture  essential  elements  of  a 
task  domain — at  the  individual,  the  team,  the  organization,  and  mission  levels  of  perspective,  to  generate 
and  manipulate  realistic  portrayals  of  mission  scenarios. 

While  advancements  in  technology  allow  more  realistic  and  cost-effective  representations  of  the 
task,  we  agree  with  Salas  and  his  associates  (Salas,  Cannon-Bowers,  &  Kozlowski,  1997)  that  computer- 
based  training  has  yet  to  be  fully  exploited  for  the  advancement  of  the  science  and  practice  of  training. 
Technology  per  se  should  not  serve  as  demonstration  of  training  improvement.  Detailed  replication  of  an 
environment,  however  complex,  will  not  ensure  psychological  validity;  indeed,  it  may  not  even  be 
necessary  (Berkowitz  &  Donnerstein,  1982;  Bowers,  Salas,  Prince,  &  Brannnick,  1992;  Dipboye  & 
Flanagan,  1979;  Driskell  &  Salas,  1992).  Certainly,  we  need  to  capture  core  characteristics  of  complex 
naturalistic  environments  if  we  mean  to  understand  and  enhance  performance  and  skill  acquisition  in 
such  situations  (Brunswick,  1956;  Hammond,  1993;  Klein,  1997;  Klein,  Orasunu,  Calderwood,  & 
Zsambok,  1993;  Klein  &  Woods,  1993;  Vicente,  1997).  The  challenge  is  to  identify  necessary  levels  of 
fidelity  to  achieve  instructional  goals. 

The  application  of  intelligent  computer-assisted  tutoring  technology  has  demonstrated  great 
success  in  enhancing  knowledge  and  skill  acquisition  of  individuals.  Computer-assisted  instruction  (CAI) 
is  a  mature  technology,  which  has  been  successfully  applied,  in  a  wide  variety  of  domains.  The  notion 
that  carefully  individualized  instruction  is  superior  to  traditional  linear  instruction  has  been  substantiated 
in  numerous  studies  (Regian  &  Shute,  1992;  1998;  Shute  &  Psotka,  1995;  Woolf,  1987).  Individualized 
instructional  software  autonomously  modifies  its  behavior  in  response  to  its  "model  of  the  student's 
current  understanding  of  the  subject  matter,"  based  on  individual  performance  indices  (VanLehn,  1986). 
Further  individualization  of  training,  as  demonstrated  by  “intelligent”  computer-based  systems,  is 
theoretically  consistent  with  models  of  cognitive  skill  acquisition  (Anderson,  1983;  Anderson,  Corbett, 
Fincham,  Hoffman,  &  Pelletier,  1992)  with  models  that  focus  on  motivational  processes  (Bandura  & 
Cervone,  1986;  Deci  &  Ryan,  1980;  Dweck,  1986;  Kanfer,  1990)  and  also  models  of  training  systems  for 
complex  behavior  (Kozlowski,  1998;  Kozlowski  &  Salas,  1997).  It  should  be  emphasized  that  the 
success  of  these  advanced  tutoring  systems  is  not  inherent  in  the  technology,  but  rather  in  the  principles 
which  guide  effective  instruction. 


1 


The  success  of  this  approach  for  individuals  is  mitigated  by  the  fact  that  individuals  almost 
always  perform  in  the  context  of  teams,  and  instructional  models  have  not  yet  addressed  the 
consideration  and  measurement  of  teamwork  performance.  Instruction  and  feedback  was  at  the 
individual  level  of  performance,  yet  in  operational  context,  the  individual  needs  to  perform  in  a  manner 
that  maximizes  the  performance  of  the  team  as  a  whole,  and  not  simply  his/her  own  performance 
outcomes.  Instructional  systems  must  include  measures  of  team  process  and  mission  outcomes  to  enable 
assessment,  coaching,  tutoring,  and/or  feedback  capabilities. 

Team  performance  is  not  nearly  as  well  understood  as  individual  performance,  and  instructional 
models  to  optimize  team  performance  are  virtually  nonexistent.  No  comprehensive  taxonomies  have 
been  developed  to  provide  systematic  dimensions  of  variation  across  team  tasks.  The  few  postulated 
models  of  team  are  generally  not  specified  sufficiently  to  support  the  generation  of  individualized 
instructional  models.  Our  general  approach  is  to  build  on  critical  developments  in  individual 
performance  and  combine  these  with  promising  approaches  for  modeling  and  optimizing  team 
performance.  First  we  describe  principles  and  characteristics  of  individualized  instruction.  We  then 
discuss  application  of  this  approach  to  instruction  of  teamwork. 

Characteristics  of  CAI  Instruction 

Virtually  all  CAI  systems  are  individualized  in  the  sense  that  they  are  self-paced,  and  many  are 
further  individualized  by  virtue  of  branching  routines  that  enable  differential  instruction.  However,  in 
branched  CAI  the  instructional  developer  must  explicitly  encode  the  actions  generated  by  all  possible 
branches,  and  there  must  be  a  finite  number  of  possible  paths  through  these  branches.  As  one  moves 
further  away  from  the  CAI  to  the  ICAI  end  of  the  continuum,  one  begins  to  see  a  more  powerful 
approach  to  individualization.  This  cognitive  approach  is  touched  on  by  Wenger  (1987)  when  he  refers 
to  explicit  encoding  of  knowledge  as  opposed  to  encoding  of  decisions  (pg.  4).  In  the  ideal  case,  ICAI 
utilizes  a  diverse  set  of  knowledge  bases  and  inference  routines  to  "compose  instructional  interactions 
dynamically,  making  decisions  by  reference  to  the  knowledge  with  which  they  have  been  provided" 
(Wenger,  1987;  pg.  5).  Table  1  characterizes  this  dimension  of  computer-assisted  instructional  systems. 
Low,  moderate,  and  high  levels  of  individualization  should  be  viewed  as  representing  areas  along  a 
continuum  of  low  to  high  individualization,  rather  than  as  discrete  categories. 

Table  1.  Levels  of  Individualization  in  Computer-Assisted  Training  Systems 


Low 

Individualization  of  Instruction 
Moderate 

High 

Knowledge 

Linear 

Branched 

Networked 

Pedagogy 

Self-paced 

Adaptive 

Intelligent 

Package 

Programmed 

Instruction 

Computer  Assisted 

Inst. 

Intelligent  CAI 

Example 

Pilot 

WPS  Tutor 

StatLady 
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ICAI  Components 


In  an  ICAI,  individualized  instruction  is  an  emergent  property  of  several  interacting  components. 
ICAI  systems  usually  consist  of  four,  sometimes  five,  components.  These  are  the  expert  module,  the 
instructional  module,  the  student  model,  the  interface,  and  often  a  device  simulation  or  other  instructional 
environment. 

The  expert  module  is  a  programmed  representation  of  expert  knowledge  in  the  target  domain 
(that  which  is  being  taught).  It  is  almost  identical  to  what  is  commonly  known  as  an  expert  system, 
except  in  this  context  it  is  often  very  articulate  (able  to  generate  some  form  of  rationale  for  its  actions) 
and  capable  of  generating  alternative  solution  paths  (rather  than  a  single  “best”  path).  The  expert  module 
brings  domain  knowledge  to  the  ICAI.  It  “knows”  how  to  perform  the  task  that  it  is  seeking  to  teach  and 
can  demonstrate  that  knowledge. 

The  instructional  module  is  a  programmed  representation  of  expert  knowledge  on  instruction  in 
the  target  domain.  Its  function  is  to  adapt  instructional  approaches  based  on  the  current  knowledge  level 
of  the  student.  While  the  expert  module  is  typically  derived  from  knowledge  representations  from  an 
expert  practitioner;  the  instructional  module  may  be  derived  from  an  expert  instructor  in  the  target 
domain,  a  general  training  specialist,  or  both. 

The  student  model  constitutes  a  repository  for  information  about  each  student  who  uses  the 
system.  It  is  a  mere  shell  at  the  beginning  of  an  initial  tutoring  session,  whereas  the  expert  and 
instructional  modules  are  generally  complete  when  the  development  of  the  ICAI  is  complete.  At  the 
beginning  of  an  initial  tutoring  session  the  student  model  stores  specific  kinds  of  information  about 
students  for  use  by  the  instructional  module.  The  student  model  maintains  updated  information  about  the 
student,  such  as  what  the  student  does  and  does  not  know,  and  any  misconceptions  indicated  by  the 
student.  The  system  is  “aware”'  of  who  is  being  taught,  and  can  make  informed  decisions  as  to  what 
should  be  instructed  next  and  the  appropriate  instructional  approach  to  be  used. 

The  interface  provides  the  methods  by  which  the  student  interacts  with  the  ICAI.  The  interface 
may  include  such  output  methods  as  computer-generated  graphics  and  text,  recorded  video  images,  or 
speech  synthesizers;  and  such  input  devices  as  a  mouse,  keyboard,  touchscreen,  joystick,  or  voice 
recognition  system.  One  important  point  about  the  interface  is  that  it  should  be  as  simple  as  possible  so 
that  learning  to  use  the  ICAI  does  not  interfere  with  learning  from  the  ICAI. 

Device  Simulation.  Many  ICAI  systems  (e.g.,  STEAMER,  IMTS/Bladefold,  Sherlock)  use  an 
embedded  computer  simulation  of  the  electrical  or  mechanical  device,  when  the  goal  is  to  understand, 
operate,  and/or  troubleshoot  the  device.  Other  ICAI  systems  teach  a  body  of  knowledge  that  is  not 
specific  to  a  particular  device,  but  use  simulations  to  provide  an  instructional  context.  For  example, 
“Smithtown”  uses  a  simulation  of  microeconomics  operating  in  a  small  town,  and  our  orbital  mechanics 
tutor  uses  a  simulation  of  orbital  dynamics. 

ICAI  and  Group  Instruction 

The  ICAI  automated  instruction  paradigm  has  been  focused  on  instructing  individually 
performed  tasks  to  individuals.  In  fact,  the  ICAI  approach  is  the  contemporary  epitome  of 
"individualized"  instruction.  However,  previous  efforts  have  acknowledged  the  relevance  of  important 
social  factors,  such  as  observational  learning,  that  may  be  very  beneficial  in  learning  certain  kinds  of 
tasks.  These  considerations  have  caused  us  to  explore  automated  instructional  systems  that  teach  small 
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Table  2.  Components  of  Intelligent  Computer-Assisted  Training  Systems 


ICAI  Components 

Function 

Expert  Module 

Representation  of  expert  knowledge, 
including  rationale  &  alternatives 

Instructional  Module 

Expert  knowledge  on  instructional 
approaches  in  target  domain 

Student  Model 

Information  regarding  student 
patterns  of  performance 

Interface 

Information  representation  and 
distribution,  Simulation  of  task 
environment 

groups  rather  than  individuals  (Shebilske  &  Regian,  1992).  Progress  has  been  significant  in  the  effort  to 
automate  group-based  instruction  for  tasks  that  are  performed  by  individuals.  Research  in  our  laboratory 
has  supported  the  notion  that  computer-based  training  (CBT)  can,  in  some  cases,  be  made  more  effective 
and  less  costly,  by  focusing  on  group  training. 

Shebilske,  Regian,  Arthur,  and  Jordan  (1992),  examined  a  dyadic  Active  Interlocked  Modeling 
(AIM)  protocol,  in  which  each  trainee  controls  half  of  a  complex  task  (i.e.,  Space  Fortress),  allowing 
each  to  learn  the  critical  connections  by  modeling  the  actions  and  reactions  of  their  partner.  Shebilske, 
Jordan,  Arthur  and  Regian  (1993)  expanded  the  dyadic  protocol  to  include  two  more  trainees,  who  rotate 
through  task  engagement  and  passive  observation.  The  results  revealed  that  four  trainees  could  learn  as 
well  as  one,  with  less  than  one-third  the  hands-on  practice,  trainer  time,  and  resources.  Similarly,  Goettl 
and  Connolly  Gomez  (1995)  using  a  desktop-flight  simulator  task  (Phoenix)  found  superior  performance 
for  subjects  who  observed  compared  to  subjects  in  a  no-observation  control  condition.  It  should  be  noted 
that  this  improvement  was  obtained  in  a  task  that  had  significant  cognitive  and  strategic  elements,  but 
was  not  obtained  for  a  task  that  was  primarily  perceptual-motor  in  nature.  Based  upon  the  results  of 
these  studies,  Shebilske  and  colleagues  concluded  that  “theories  of  automatization  will  have  to  be 
expanded  to  incorporate  the  role  of  observational  learning”  (Johnson,  Regian  &  Shebilske,  1994,  p.  6). 
So  far  results  indicate  that  the  cost-effectiveness  of  group-based  observational  learning  may  depend  on 
the  degree  to  which  task  demands  are  cognitive/perceptual  as  opposed  to  psychomotor/motor  in  nature. 
Systematic  investigation  should  verify  this  interaction  and  identify  other  characteristics  that  would  enable 
more  accurate  evaluation  of  this  approach. 

Group-based  discussion  is  another  group  instructional  technique,  which  should  be  further 
studied.  We  have  explored  discussion  groups  using  the  Space  Fortress  task.  Prislin,  Jordan,  Worchel, 
Tschan-Semmer  and  Shebilske  (1996)  found  that  discussion  groups  resulted  in  improved  performance, 
particularly  for  females.  Male/female  differences  have  been  noted  for  different  cognitive  skills,  group 
problem  solving  strategies,  team  effectiveness  in  different  types  of  tasks,  and  communication  style  in 
general  (Matheson,  1991;  Taps  &  Martin,  1990).  These  differences  should  be  identified  to  further 
understand  the  nature  of  these  differences  and  design  interventions  to  optimize  team  performance. 
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Group-based  instruction  can  also  affect  the  degree  to  which  trainees  will  be  collaborative  or 
competitive.  Collaboration  and  competitiveness  can  be  induced  through  manipulation  of  features  such 
as: 


-  Private  versus  public  feedback  (Nordstrom,  Lorenzi,  &  Hall,  1991;  Pritchard,  Jones,  Roth, 

Stuebing  &  Ekeberg,  1988;  1989) 

-  Performance  versus  mastery  goals  (Dweck,  1986;  Kozlowski,  1996;  Kozlowski  et  al.,  In  press) 

-  Reward  systems  (Latham  &  Locke,  1991;  Locke  &  Latham,  1990;  Pritchard  et  al.,  1988;  1989) 

-  Degree  of  interdependence  (Mitchell  &  Silver,  1990;  Saavedra,  Early  &  Van  Dyne,  1993) 

Prislin  et  al.  (1996)  examined  the  effects  of  competition  on  skill  acquisition.  Competition 
produced  beneficial  effects  on  skill  acquisition  when  it  was  introduced  late  in  training  and  inhibitory 
effects  when  it  was  introduced  early  in  training.  This  is  consistent  with  the  literature  on  mastery  versus 
performance  goals  (Kozlowski,  1996;  Dweck,  1986).  The  study  also  showed  that  effectiveness  of 
individual  versus  dyadic  training  was  moderated  by  the  level  of  interaction  anxiety  of  the  trainees. 
Trainees  with  high  anxiety  demonstrated  higher  performance  in  individual  conditions,  while  subjects  low 
in  interaction  anxiety  had  higher  performance  in  dyads. 

These  findings  confirm  the  importance  of  social  factors  in  skill  acquisition  and  argue  for  a  new 
generation  of  1CA1  systems  that  are  oriented  around  a  group  pedagogy.  Such  a  pedagogy  has  been 
demonstrated  as  more  effective  than  individual-oriented  pedagogy  and  may  in  fact  generalize  to  training 
situations  with  an  emphasis  on  declarative  knowledge  as  opposed  to  the  training  of  complex  psychomotor 
or  motor  skills.  In  fact,  by  intelligently  combining  observational  learning,  discussion  groups,  and 
competition,  it  may  be  possible  to  develop  an  1CA1  based  on  group  pedagogy  that  is  superior  to  one  based 
on  individual  pedagogy.  It  is  clear  research  is  needed  in  this  area,  to  identify  boundary  conditions,  task 
characteristics,  and  individual  differences  which  most  influence  the  effectiveness  of  individual  versus 
group-based  instruction. 

ICAI  and  Teamwork  Instruction 

We  expect  that  principles  of  skill  acquisition  and  instruction  validated  at  the  individual  level  will 
generalize  to  the  individual  team  member,  once  teamwork  task  demands  are  identified  for  each  team 
member.  This  approach  will  enable  systematic  generation  and  evaluation  of  teamwork  training 
principles-to  predict  what  should  be  trained,  to  whom,  in  what  sequence,  using  what  type  of  instructional 
delivery,  in  what  kind  of  context.  Here,  we  describe  progress  towards  development  of  a  taxonomy  of 
teamwork  tasks  and  associated  cognitive  demands. 

As  discussed  above,  there  are  several  factors  that  influence  skill  acquisition  in  group  instruction, 
even  when  all  participants  are  learning  the  same  task— observational  learning,  discussion  groups, 
cooperative  versus  competitive  dynamics,  and  social  factors.  When  we  consider  the  application  of  1CA1 
systems  to  the  training  of  work  teams,  the  situation  becomes  even  more  challenging.  Work  teams  are 
often  comprised  of  members  with  diverse  expertise,  who  must  coordinate  information,  decisions,  and 
actions  in  order  to  achieve  a  mutual  goal. 

As  described  above,  in  our  program  of  research  we  have  investigated  the  usefulness  of  teaching 
small  groups  to  perform  tasks  that  in  the  end  will  be  performed  individually.  We  believe,  however,  that 
pedagogy  designed  for  individuals  can  be  effectively  applied  for  the  training  of  team  performance,  once 
key  issues  are  addressed.  Individualized  instruction  is  often  developed  for  relatively  static  task 
environments,  such  as  the  operation  and  maintenance  of  stable  systems  (Lesgold,  Lajoie,  Bunzo,  & 
Eggan,  1992).  The  knowledge  required  for  such  tasks  can  be  captured  through  identification  of  key  “if- 
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then”  and  “how  to”  statements.  Procedures  can  be  specified  in  advance,  and  team  members  need  only  to 
follow  these  procedures  to  attain  coordinated  performance.  However,  team  performance  often  occurs  in 
dynamic  environments-indeed  the  pervasive  use  of  teams  by  organizations  is  to  a  large  extent  in 
response  to  the  need  for  more  flexible  and  adaptive  structures. 

The  primary  challenge  here  is  to  develop  instruction  to  enhance  teamwork  performance  in 
complex  and  challenging  situations.  We  must  be  able  to  identify  the  knowledge  and  skills  that  comprise 
effective  teamwork  in  different  settings,  particularly  those  related  to  success  in  ambiguous  and  dynamic 
environments.  Our  approach  builds  upon  our  TRAIN  (Training  Research  for  Automated  INstruction) 
model  for  individuals,  extending  the  model  to  include  teamwork  functions.  A  taxonomic  approach  is 
used  to  characterize  teamwork  functions  and  prerequisite  knowledge  and  skills. 

Researchers  at  the  TRAIN  laboratory  have  developed  and  validated  a  theory-based  approach  to 
characterizing  tasks  that  are  performed  by  individuals.  The  approach  has  proven  effective  for  the  design 
of  training  for  individually  performed  tasks.  The  approach  rests  on: 

(1)  a  General  Learning  Theory  that  specifies  dimensions  of  knowledge  and  skill  and 
corresponding  characteristics  of  skill  acquisition, 

(2)  a  Cognitive  Taxonomy  that  allows  reliable  decomposition  of  tasks  into  knowledge/skill  types 
that  support  task  performance, 

(3)  a  General  Instructional  Model  that  specifies  how  acquisition  of  these  knowledge/skill  types 
can  be  optimized,  and 

(4)  a  set  of  Criterion  tasks  that  allow  us  to  systematically  conduct  studies  on  human  acquisition 
of  these  knowledge/skill  types. 


Figure  1.  Instructional  Engineering  for  Individually  Performed  Tasks 

We  are  now  attempting  to  scale  our  approach  to  individualized  instructional  engineering  up  to 
team-level  performance.  At  the  team  level,  we  proposed  to  develop  theory  and  taxonomies 
correspondent  to  our  individual  approach,  to  include: 

(1)  a  general  Theory  of  Teamwork  Performance 

(2)  a  Cognitive  Taxonomy  of  Teamwork  Demands 

(3)  a  general  Team  Training  Model 

(4)  a  set  of  Team  Criterion  Tasks 
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Any  approach  to  team  training  must  deal  with  the  relationship  between  individual  performance 
and  team  performance.  First,  we  must  specify  distinctions  among  individual  and  team-level  performance 
constructs.  What  are  the  knowledge  and  skills  that  underlie  successful  team  performance,  and  to  what 
degree  are  they  generalizable  across  categories  of  team  tasks?  To  address  these  questions,  taxonomies 
must  be  specified  to  generate  a  systematic  approach  relating  aspects  of  team  functions  with  aspects  of 
task  knowledge  and  skill  requirements.  Boundary  conditions  should  be  drawn  from  a  taxonomy  of  team 
task  characteristics.  From  these  dimensions,  propositions  are  made  with  regard  to  development  of 
instructional  content,  drawn  from  extensive  accomplishments  in  development  of  individualized  training. 


Figure  2.  Instructional  Engineering  for  Teamwork  Tasks 


Once  theoretical  constructs,  relationships,  and  boundary  conditions  are  identified,  systematic 
investigations  regarding  optimal  training  of  teamwork  functions  can  be  initiated.  For  example,  should  we 
train  individual  skills  outside  of  the  team  task  context,  or  simultaneously  with  team  skills?  If  skills  are 
taught  as  part-tasks,  which  skills  should  be  taught  separately,  and  in  what  sequence?  Are  there  generic 
teamwork  skills  that  can  be  taught  using  a  synthetic  task?  What  level  of  fidelity  in  synthetic  tasks  is 
required  for  effective  training?  We  believe  that  a  useful  characterization  of  team  task  performance  should 
emphasize  individual  task  performance  models  for  each  member  of  the  team.  Team  performance  is 
ultimately  an  algorithm  comprised  of  elements  of  individual  performance. 

ft  has  been  stated  that  an  expert  team  is  more  than  just  a  sum  of  its  members  (Salas  et  al.,  1997); 
nonetheless,  it  is  still  the  case  that  overall  team  performance  is  effectively  the  interplay  of  individuals 
performing  in  a  team  context.  We  expect  team  members  high  in  task  expertise  who  nevertheless  perform 
poorly  as  a  team  are  lacking  teamwork  skills,  and  that  these  skills  can  be  trained.  Skill  acquisition  is  a 
individual-level  phenomenon,  which  can  be  represented  at  the  team  level  to  compare  teams,  but  is  a 
function  of  the  individual.  Instructional  concepts  validated  at  the  individual  level  should  transfer  to  the 
training  of  teamwork  skills. 

This  is  not  to  say  teamwork  should  be  trained  as  an  individual  task,  although  much  of  it  could  be 
accomplished  given  advancements  in  computer-based  simulations.  Simulations  can  now  present  a 
complex  environment,  filled  with  computer-generated  hostile  and  friendly  “entities”  and  “advisors” 
(Elliott  et  al.,  in  review;  Schiflett  &  Elliott,  2000).  Trainees  can  interact  with  computer-generated  team 
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members  that  can  vary  in  level  of  expertise,  thus  providing  a  high  level  of  control  and  reduction  of  error. 
On  the  other  hand,  there  is  no  substitute  for  interaction  with  actual  people,  when  studying  sources  of 
error  arising  from  the  ability  to  “deal”  with  various  personality  characteristics  and  achieve  effective 
group  problem-solving,  conflict  resolution,  and  negotiation  skills. 

Performance  requirements  that  apply  only  to  individual  team  members  (individual  task  expertise) 
will  be  modeled  at  the  individual  level,  using  our  existing  tools.  Team  level  performance  requirements, 
such  as  information  exchange,  performance  monitoring,  resource  allocation,  and/or  group  problem 
solving  will  be  modeled  at  the  individual  level  and  as  team-level  performance  constructs.  While  much  is 
stated  about  team  training  and  team  mental  models,  it  is  useful  to  maintain  the  distinction  between 
individual  skill  acquisition,  expertise  and  performance,  versus  team-level  constructs  which  are  truly  a 
function  of  the  team  as  a  whole,  such  as  morale,  cohesiveness,  coordination,  and  overall  performance 
outcomes.  There  are  several  lessons-leamed  with  regard  to  conceptualization  and  measurement  of  team- 
level  constructs,  discussed  at  length  elsewhere  (James,  1982;  Jones,  1974). 

We  draw  upon  the  substantial  work  accomplished  by  Salas  and  his  colleagues  (Brannick,  Roach, 
&  Salas,  1993;  Cannon-Bowers,  Bums,  Salas,  &  Pruitt,  1998;  Cannon-Bowers  &  Salas,  1998;  Salas, 
Bowers,  &  Cannon-Bowers,  1995;  Salas,  Cannon-Bowers,  &  Blickensderfer,  1997;  Salas,  Cannon- 
Bowers,  &  Johnston,  1997;  Salas,  Dickenson,  Converse,  &  Tannenbaum,  1992;  Swezey  &  Salas,  1992) 
for  our  initial  starting  point.  Consistent  with  his  model  of  team  performance,  we  distinguish  team 
performance  as  a  function  of  taskwork  and  teamwork.  We  then  propose  a  taxonomy  of  core  teamwork 
functions  that  support  management  of  team  member  interdependencies.  Our  current  plan  is  to  model 
team  level  team-task  performance  using  a  single  instantiation  of  the  Team  Performance  Model  (currently 
version  1.0),  and  to  model  individual  team-task  performance  using  multiple  instantiations  of  the  TRAIN 
cognitive  model  (currently  version  2.6). 


Figure  3.  Instructional  Engineering:  Teamwork  and  Taskwork 


To  provide  a  starting  point  for  team-level  instructional  engineering,  we  specify  (a)  an  initial 
Team  Performance  Theory  and  (b)  associated  taxonomies  to  characterize  teamwork  functions,  teamwork 
knowledge  and  skill  requirements,  and  team  task  characteristics.  These  hypothesized  constructs  are  our 
initial  attempt  to  decompose  teamwork  tasks  into  discriminable  functions  that  support  task  performance, 
which  can  be  generalized  to  teams  in  general,  and  to  quantify  dimensions  of  teamwork  task  complexity. 
They  constitute  our  first  attempt  at  modeling  team-level  task  performance.  Our  initial  taxonomies  of 
teamwork  functions  and  teamwork  demands  are  described  in  the  following  sections. 

To  provide  a  starting  point  for  team-level  instructional  engineering,  we  specify  (a)  an  initial 
Team  Performance  Theory  and  (b)  associated  taxonomies  to  characterize  teamwork  functions,  teamwork 
knowledge  and  skill  requirements,  and  team  task  characteristics.  These  hypothesized  constructs  are  our 
initial  attempt  to  decompose  teamwork  tasks  into  discriminable  functions  that  support  task  performance, 
which  can  be  generalized  to  teams  in  general,  and  to  quantify  dimensions  of  teamwork  task  complexity. 
They  constitute  our  first  attempt  at  modeling  team-level  task  performance.  Our  initial  taxonomies  of 
teamwork  functions  and  teamwork  demands  are  described  in  the  following  sections. 

Teamwork  Task  Demands:  A  Functional  Approach 

In  dynamic  situations,  teamwork  becomes  vital:  to  manage  interdependencies  team  members 
must  maintain  awareness  of  changes  in  strategic  information,  gained  from  a  variety  of  sources.  Team 
members  are  often  in  communication  with  different  individuals,  teams,  organizational  entities  (such  as 
coiporate  headquarters,  customers,  and/or  competitors),  and  computer/equipment/decision  aids.  In 
addition,  team  members  may  have  different  areas  of  expertise,  may  be  working  on  different  aspects  of  the 
team  goal,  and  may  be  located  in  different  geographic  locations.  As  a  result,  team  members  will  have 
different  interpretations  of  their  task  environment.  This  information  should  be  interpreted  and  shared  in 
order  to  identify  inconsistencies,  generate  alternate  interpretations,  and  discover  emerging  problems. 

Team-based  training  has  been  shown  to  be  an  effective  means  to  train  knowledge  and  skill 
requirements  for  efficient  coordination  of  information  (Cannon-Bowers  et  al.,  1998;  Kozlowski  et  al, 
1998;  Swezey  &  Salas,  1992).  Similarly,  it  is  expected  that  team-based  training  can  enhance 
development  of  adaptive  expertise  within  dynamic  contexts.  In  fact,  there  is  quite  a  bit  of  overlap  in 
investigations  of  complex  naturalistic  performance  and  team  performance,  for  both  are  critical  to  the 
understanding  of  expert  performance  in  context  (Beach,  Chi,  Klein,  Smith,  &  Vicente,  1997;  Brehmer, 
1992;  1998;  Cannon-Bowers  &  Bell,  1997;  Coovert  et  al.,  1995;  Drillings  &  Serfaty,  1997;  Howell, 
1997;  Klein,  1993;  1997;  Kleinman  &  Serfaty,  1989;  Klein  &  Woods,  1993;  Kozlowski,  1996; 
Kozlowski  et  al.,  1999). 

Our  goal  in  developing  the  Teamwork  Taxonomy  is  to  categorize  and  describe  team  tasks  at  a 
level  of  analysis  that  supports: 

1 )  Abstract  descriptions  of  task  performance 

2)  Predictions  about  instructional  approaches  that  will  optimize  task  performance 

3)  Generalizations  about  instructional  approaches  that  will  work  across  team  tasks 

4)  Recommendations  about  team  task  structure  and  team  task  performance  contexts 

5)  Recommendations  about  team  member  selection 

A  taxonomic  approach  is  essential  for  systematic  investigation  of  team  performance  dynamics 
and  skill  acquisition.  Critical  boundary  conditions  need  to  be  identified  as  part  of  the  theory-building 
process.  It  is  evident  that  teams  span  a  diverse  array  of  functions.  Several  taxonomies  have  provided 
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some  key  distinguishing  factors  among  these  teams  (Cannon-Bowers,  Oser,  &  Flanagan,  1992;  Fleishman 
&  Zaccarro,  1992;  Salas  et  al.,  1992;  1995;  Sundstrom,  De  Meuse,  &  Futrell,  1990;  Swezey  &  Salas, 
1992). 


Existing  taxonomies  of  ability  requirements  and  task  characteristics  were  reviewed,  and  each  is 
useful  for  its  stated  puipose.  Flowever  a  taxonomy  is  needed  that  can  be  used  to  distinguish  teams  on  the 
basis  of  the  degree  and  type  of  teamwork  needed  for  successful  performance.  This  taxonomy  would  be 
used  to  (a)  distinguish  types  of  teams,  (b)  identify  teamwork  training  needs,  and  (c)  provide  a  framework 
for  teamwork  performance  assessment. 

Our  teamwork  taxonomy  must  effectively  categorize  the  demands  that  performance  of  any  team 
task  makes  on  the  team,  and  must  allow  any  task  to  be  described  in  terms  of  both  team  and  individual 
capabilities  that  support  performance  in  the  task.  Review  of  existing  taxonomies  resulted  in  a  long  and 
varied  list  of  many  types  of  behaviors  (e.g.,  “assisting  others,”  “requesting  help,”  “exchanging 
information”),  functions  (e.g.,  “monitor  performance  of  others,”  “coordination  of  tasks”),  and  skills  (e.g., 
skills  which  facilitate  resolution  of  interpersonal  and  task  decision  conflicts).  Many  of  the  constructs  in 
existing  taxonomies  appeared  as  if  they  may  be  useful  for  some  types  of  teams  but  not  others.  For 
example,  it  is  intuitive  to  expect  that  collaboration  and  conflict  resolution  skills  would  be  needed  in 
teams  that  have  a  consensual  decision  process,  but  perhaps  not  for  teams  where  decisions  must  be  made 
quickly  by  one  individual.  In  the  same  way,  monitoring  the  performance  of  other  team  members  may  be 
critical  for  some  teams,  and  detrimental  to  others  (when  the  information  is  not  necessary  and  time  is 
short).  It  seems  clear  that  critical  aspects  of  teamwork  performance  will  depend  on  contextual  factors 
such  as  task  demands,  information  characteristics,  and  team  structure.  Flow  then  can  one  systematically 
approach  various  kinds  of  team  settings  and  identify  teamwork  requirements?  We  began  with  a 
functional  definition  of  teamwork,  which  can  generalize  across  all  types  of  teams  and  settings. 

What  is  teamwork?  We  began  with  the  definition  of  teams.  Teams  are  distinguished  from 
groups  in  general  by  a  common  puipose  or  goal,  performed  by  interdependent  team  members  (Swezey  & 
Salas,  1992).  From  this  definition  we  derive  a  core  definition  of  teamwork,  as  contrasted  with  individual 
taskwork:  the  fundamental  function  of  teamwork  is  the  effective  managing  of  interdependencies  to 
accomplish  team  goal. 

The  sheer  diversity  of  teams  makes  it  difficult  to  identify  core  components  of  teamwork. 
Consider  the  types  of  teams  described  in  existing  taxonomies.  What  can  these  teams  have  in  common  in 
terms  of  how  to  work  more  effectively  together?  One  core  dimension  that  characterizes  teams  in  general 
is  the  type  and  degree  of  interdependence  among  team  members  (Saavedra,  et  al.,  1993;  Swezey  &  Salas, 
1992).  Teams  are  distinguished  from  groups  in  general  by  a  common  purpose  or  goal,  performed  by 
interdependent  team  members  (Salas,  Dickinson,  Converse,  &  Tannenbaum,  1992).  From  this  definition 
we  derive  a  core  definition  of  teamwork:  The  fundamental  function  of  teamwork  is  the  effective 
managing  of  interdependencies  to  accomplish  team  goal.  Based  on  this  focus  on  team  member 
interdependencies,  we  generated  an  initial  taxonomy  to  distinguish  team-level  functioning  based  on  type 
and  degree  of  interdependence  among  team  members.  The  purpose  of  this  generic  taxonomy  is  to  serve 
as  an  overarching  framework  to  distinguish  teams  in  general,  across  performance  domains,  with  regard  to 
the  nature  and  extent  of  teamwork  demands. 

A  fundamental  aspect  of  the  demand  for  teamwork  is  the  degree  to  which  interdependencies  exist 
among  team  members.  To  ascertain  this  one  must  define  the  nature  by  which  team  members  may  be 
interdependent  in  such  a  way  that  can  be  generalized  among  various  types  of  teams.  Assuming  the 
essence  of  teamwork  is  the  managing  of  team  member  interdependencies,  we  identified  two  core 
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dimensions  with  regard  to  nature  and  extent  of  interdependencies,  that  of  (a)  coordination  complexity 
(static)  and  (b)  adaptive  replanning  (dynamic).  Each  core  function  can  be  analyzed  to  identify  tasks  and 
task  sequences  necessary  to  accomplish  functional  goals  (Schiflett  &  Elliott,  2000). 

Another  fundamental  dimension  of  teamwork  task  demand  that  is  expected  to  distinguish 
knowledge  and  skill  requirements  and  training  strategy  is  the  nature  of  the  decision  process  used  to 
accomplish  each  aspect  of  teamwork.  The  team  decision  making  process  may  be  hierarchical,  when  there 
is  a  team  leader  who  makes  a  final  decision,  or  it  may  be  dependent  on  consensus.  If  there  is  a  strict 
hierarchy  in  the  team  structure,  such  as  in  most  military  teams,  one  achieves  efficiency  in  high-tempo 
situations,  but  the  team  and  particularly,  the  team  leader,  must  be  trained  to  perform  effectively  in  this 
context.  If  the  team  decision  making  is  more  consensual,  negotiation  and  conflict  resolution  skills 
become  important  and  different  training  content  and  decision  aiding  tools  would  be  indicated.  This  is 
particularly  true  if  the  team  is  comprised  of  specialists  as  opposed  to  generalists. 

There  are  other  dimensions  that  can  be  related  to  training  content/delivery,  but  each  would  be 
associated  with  a  subtask  required  to  achieve  underlying  teamwork  functions  in  a  particular  task  setting 
and  thus  are  not  considered  core.  We  begin  with  a  focus  on  three  core  dimensions  characterized  in  all 
team  settings.  They  are  expected  to  distinguish  requirements  for  effective  teamwork  and  training 
content/delivery: 

1)  Degree  of  Coordination  Complexity 

2)  Degree  of  Dynamic  Replanning 

3)  Type  of  Decision  Process — Hierarchical  Versus  Consensus 

Once  the  performance  domain  has  been  analyzed  for  these  core  teamwork  task  demands,  we  can 
generate  hypotheses  as  to  knowledge,  skills,  and  abilities  required  for  successful  performance  for  both 
aspects,  thus  identifying  requirements  for  overall  team  performance. 

Determining  Cognitive  Demands:  Cognitive  Taxonomy  of  Teamwork 

The  model  described  above  provides  a  core  conceptualization  of  teamwork,  based  on  three 
independent  dimensions  underlying  all  team  tasks.  Any  team  task  can  be  characterized  along  these 
dimensions.  Each  dimension  has  implications  for  teamwork  skill  requirements  and  team  training 
content/delivery.  From  this  framework,  functions  can  be  identified  for  a  particular  team  setting,  as  a  first 
step  in  identifying  specific  subtasks  and  their  corresponding  cognitive/skill  requirements.  The  final  level 
of  detail  and  specificity  will  depend  on  the  purpose  of  data  collection  (e.g.,  identification  of  criterion 
measures,  training  requirements,  decision  support  tools). 

Our  approach  to  performance  engineering  requires  identification  of  underlying  cognitive 
demands.  For  example,  a  team  task  may  have  a  high  requirement  for  coordination  achieved  through 
performance  monitoring  (of  self  and  team  members),  information  retrieval,  and  information  exchange. 
The  optimal  instructional  method  to  enhance  these  tasks  would  depend  on  the  nature  of  the  task. 
Performance  monitoring  in  one  team  setting  may  be  straightforward  and  easily  acquired  through 
repetition  and  practice.  A  different  team  setting  may  demand  much  more  skill  in  information  processing 
and  require  a  different  approach  to  training  generalizable  skills. 

Once  teamwork  functions  and  tasks  are  identified,  they  can  be  further  analyzed  with  regard  to  type  and 
degree  of  cognitive  demand.  For  example,  the  problem  solving  required  by  teams  with  high  levels  of 
dynamic  replanning  can  be  relatively  easy  (certain  information,  distinct  decision  rules)  or  challenging 
(uncertain  information,  ambiguous  rules  of  engagement).  In  addition,  time  pressure  can  alter  the  type  of 
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decision  process  used,  from  careful  deliberation  to  a  more  satisfying,  recognition-based  process.  The 
following  table  characterizes  the  cognitive  demand  of  tasks  based  on  information  and  decision  process 
from  simple  to  complex. 


Table  3.  Cognitive-Based  Taxonomy  of  Teamwork 
Task  Components  1  Simple  2  3  Complex  4 


INFORMATION 

CONCRETE 

CONSISTENT 

CONTINGENT 

FUZZY 

Dynamic 

Information 

No  Interpretation 

Consistently 

Situationally 

Ambiguous 

Requirements 

Required 

Interpreted 

Consistent 

Subjective 

Information 

Interpretations 

Interpretations 

DELIBERATION 

FACTS 

CONCEPTS 

SCHEMATA 

MENTAL 

Prerequisite 

MODELS 

Shared 

Linear 

Fully 

Situationally 

Knowledge  and 

Information 

Specified 

Consistent 

Fuzzy,  Complex 

Interaction 

Flow 

Branches 

Branches 

Procedures 

Procedures 

Contingencies 

RESOLUTION 

EASY  RULES 

RULES 

CONTINGENCIES 

EXPERT 

Decision  Rule 

JUDGMENTS 

Specificity 

Simple 

Consistent 

Situationally 

Difficulty 

Fully 

Decision 

Consistent 

Fuzzy 

Specified 

Rules 

Decision 

Decision 

Rules 

Rules 

ACTION 

BASIC 

BASIC  SKILLS 

SKILLED 

EXPERT 

ABILITIES 

PERFORMANCE 

PERFORMANCE 

Skill  level 

Specified 

Existing  Skills 

Procedures 

Complex  Integrated 

Complex 

Low  Coordination 

Easily 

Can  be  trained 

Integrated 

Automated 

to  Automaticity 

Dynamic 
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This  taxonomy  portrays  facets  of  team  information  processing  and  decision  making  along  a 
dimension  of  cognitive  complexity.  It  is  derived  from  a  taxonomy  generated  and  validated  at  the 
individual  level,  at  the  TRAIN  laboratory.  The  taxonomy  describes  aspects  of  information  processing 
(i.e.,  information  retrieval  and  deliberation),  decision  making,  and  skill  requirements.  It  also  provides  the 
basis  by  which  each  of  these  functions  can  be  categorized  with  regard  to  complexity.  Once  teamwork 
tasks  are  identified,  tasks  can  be  categorized  along  this  dimension,  and  propositions  established  at  the 
individual  level  from  prior  research  can  be  generalized  to  team-level  performance. 

While  aspects  of  teamwork  may  be  fairly  consistently  associated  with  levels  of  complexity  and 
difficulty  (e.g.,  a  high  degree  of  coordination  complexity  or  dynamic  replanning  would  indicate  high 
cognitive  demand),  it  is  necessary  that  all  teamwork  tasks  and  subtasks  within  a  particular  context  be 
evaluated  along  this  dimension.  It  should  be  noted  that  a  particular  type  of  team  task  can  still  vary 
widely  in  complexity,  depending  on  context,  goals  and  expectations.  As  a  simple  example,  one  can 
easily  distinguish  differences  along  each  aspect  for  different  basketball  teams.  Consider  the  differences 
in  cognitive  and  skill  requirements  to  perform  well  in  a  volunteer  company-sponsored  team  versus  the 
NCAA.  The  goals,  rules,  and  strategies  may  apply  to  both  sets  of  teams,  but  training  and  performance 
goals  are  substantially  different. 


DISCUSSION 

Our  goal  is  to  investigate  skill  acquisition  and  performance  within  more  complex,  dynamic, 
team-based  settings.  To  accomplish  this,  we  rely  on  advanced  techniques  for  representation, 
characterization,  and  instruction  of  expert  knowledge.  Intelligent  tutoring  based  on  cognitive  science  has 
repeatedly  proven  itself  for  individual  skill  acquisition  across  a  variety  of  performance  domains  (Regian 
&  Wolfe,  1999).  The  transition  to  a  more  team-based  approach  will  require  (a)  explication  of  teamwork 
tasks,  based  on  a  broad,  generalizable  taxonomy  of  teamwork,  (b)  knowledge  elicitation  techniques  to 
enhance  capture  of  teamwork  demands  in  a  particular  setting,  (c)  development  of  synthetic  environments 
with  core  characteristics  which  enable  skill  acquisition  of  teamwork  tactics,  strategies,  and/or  problem 
solving,  and  (d)  modeling  approaches  which  enable  explication  of  an  expert  model  and  student  models  at 
both  individual  and  team  levels  of  analysis. 

This  becomes  particularly  challenging  in  naturalistic  environments  characterized  by  complexity, 
ambiguity,  interdependency  of  events  among  team  members,  and  interdependency  of  events  over  time. 
These  challenges  are  well  discussed  elsewhere  (Beach,  et  al.,  1997;  Beach  &  Lipshitz,  1993;  Brehmer, 
1992;  1998;  Cannon-Bowers  &  Salas,  1998;  Drillings  &  Serfaty,  1997;  Lipshitz  1993;  1997;  Rasmussen, 
1993;  Zsambok,  1997).  Thus,  we  began  with  a  focus  on  this  kind  of  setting,  with  the  goal  of  creating 
taxonomies,  which  distinguish  relevant  boundary  conditions  for  investigations.  We  needed  to  articulate 
this  kind  of  setting  in  a  systematic  fashion,  based  on  existing  frameworks  of  cognition,  information 
processing,  and  cognitive  demand. 

A  first  step  in  this  process  was  the  identification  and  review  of  existing  team  research  findings 
relevant  to  highly  complex  and  dynamic  operational  environments.  Current  knowledge  of  teamwork 
performance  has  benefited  greatly  from  the  contributions  of  several  researchers  over  the  past  15  years, 
resulting  in  identification  of  many  methodological  and  psychometric  issues  particular  to  team-based 
research  (Hollenbeck  et  al.,  1994;  1996;  1998;  Salas  et  al.,  1992,  1995,  1997;  Serfaty,  Entin,  &  Johnston, 
1998).  For  example,  while  teamwork  is  indeed  based  on  the  interplay  of  individuals,  the  measure  of 
teamwork  is  not  necessarily,  or  even  commonly,  based  on  the  sum  of  individual  performance.  Team 
success  could  be  a  function  of  the  best  team  member,  or  of  the  worst. 
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The  key  is  conceptualization  and  measurement  of  the  “interplay”  itself,  in  various  contexts. 
Researchers  have  generated  various  taxonomies,  classification/coding  schemes,  and  theoretical  models 
involving  constructs  such  as  team  coordination,  cohesiveness,  collective  efficacy,  shared  mental  models, 
hierarchical  decision  processes,  and  team  situation  awareness.  However,  empirical  validation  of 
theoretical  predictions  regarding  operational  performance  has  been  hampered  by  a  lack  of  similarity 
between  highly  controlled  laboratory  team  tasks  and  the  dynamic  complexities  found  in  realistic  settings. 
This  is  not  to  say  that  findings  generated  from  laboratory  settings  are  irrelevant;  but  rather  that  advanced 
platforms  can  be  used  to  demonstrate  generalizability  of  theoretical  propositions  to  more  naturalistic 
expert-based  performance  domains. 

This  paper  described  our  general  approach  to  investigate  team  training  using  principles  of  1CA1. 
First,  we  described  theory  and  principles  related  to  intelligent  tutoring  systems.  Then,  we  discussed  the 
nature  of  teamwork  and  proposed  a  functional  taxonomy  of  teamwork  to  enable  systematic  classification 
of  team  task  demands  and  dimensions  of  teamwork  performance.  This  taxonomy  proposed  two  core 
functions  of  teamwork,  one  static  (coordination  complexity)  and  one  dynamic  (adaptive  replanning)  and 
another  fundamental  dimension  of  team  decision  making  process  (hierarchical  versus  consensual-based 
processes).  These  dimensions  are  expected  to  distinguish  teams  by  teamwork  skill  requirements,  and 
assist  in  identification  of  training  content  /  delivery. 

Once  type  and  level  of  teamwork  tasks  are  identified,  they  are  further  analyzed  using  cognitive 
analysis  techniques  to  determine  level  of  cognitive  complexity  for  each  team  task  and  subtask.  To 
accomplish  this,  we  delineated  a  cognitive  taxonomy  of  teamwork,  based  on  an  information-processing 
model  of  cognitive  demand  and  cognitive  function.  We  expect  that  principles  of  instruction  based  on  this 
cognitive  taxonomy,  proven  to  be  successful  at  individual-level  instruction,  will  transfer  to  the  training  of 
teamwork. 

Systems  are  currently  being  developed  to  facilitate  research  in  team  training  and  performance. 
For  example,  synthetic  task  environments  were  developed  to  capture  essential  higher-level  functioning  of 
weapons  directors  who  work  aboard  the  Airborne  Warning  and  Control  System  (AW ACS)  aircraft 
(Chiara  &  Stoyen,  1997;  Elliott  et  al.,  2000a,  b;  Mahan,  1998;  Hess  et  al.,  1999;  Hess  &  Elliott,  2000; 
Schiflett  &  Elliott,  2000).  These  synthetic  environments  are  simpler  than  the  actual  aircraft  systems, 
which  have  numerous  specific  switch  actions  to  accomplish  a  variety  of  actions.  Instead,  the  synthetic 
tasks  may  be  accomplished  by  mouse  clicks  and  menu  selections,  a  task  already  familiar  to  anyone  who 
uses  a  PC.  We  presented  two  such  systems  to  operational  experts.  While  the  two  systems  were  different 
with  regard  to  specific  actions,  the  experts  quickly  recognized  their  fundamental  tasks  and  had  no 
problem  adjusting  to  either  interface.  Procedures  regarding  information  retrieval,  communication,  and 
decision  making  were  quickly  learned,  in  about  five  minutes.  Participants  were  soon  immersed  in  higher- 
level  tactics  and  decision  making. 

The  next  challenge  is  the  representation  of  knowledge  and  performance  within  these  team-based 
systems,  particularly  in  highly  demanding  and  complex  settings.  In  order  to  insert  1CA1  technology  in 
these  systems,  we  need  quantitative  representations  of  expert  and  student  performance.  An  initial  start  in 
this  area  has  been  accomplished,  within  one  of  the  AWACS-based  systems.  Intelligent  agent  technology 
was  utilized  to  build  a  decision-aiding  capability  that  provides  team  members  with  recommendations  for 
particular  decision  events.  These  recommendations  are  based  on  representations  of  each  type  of  decision 
task,  thus  comprising  an  expert  model  for  those  tasks.  However,  this  system  is  not  an  1CA1  system.  It  is 
more  strictly  a  decision  aid  that  can  be  utilized,  or  not.  While  additional  programming  is  necessary  to 
provide  tutoring  capability,  progress  toward  that  goal  has  been  demonstrated.  The  rationale  for 
recommendations  is  not  available  at  the  time  it  is  generated;  however,  each  decision  event  is  logged  in  a 
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data  file  along  with  the  recommendation,  the  rationale,  and  the  action  taken.  Further  refinements  would 
provide  more  feedback  on  each  decision  event  as  it  occurs. 

Again,  the  challenge  in  building  these  systems  is  the  representation  of  performance  and 
knowledge  in  these  complex  settings.  Many  contributions  have  been  made  recently  which  relate  to 
conceptualization  and  quantification  of  dynamic  C3  performance  (Benson  et  al.,  1998;  Chin,  Sanderson, 
&  Watson,  1999;  Coovert  &  Riddle,  1999;  Kleinman  et  al.,  1992;  Lee  &  Carley,  1998;  Levchuk,  Luo, 
Levchuck,  Pattipati,  &  Kleinman,  1999;  Levchuk,  Pattipati,  &  Kleinman,  1998;  1999;  Levis  &  Vaughan, 
1999;  Paley,  Levchuk,  Serfaty,  &  MacMillan,  1999).  One  or  more  of  these  approaches  may  be  the  means 
by  which  expert  /  student  performance  can  be  represented  and  interpreted. 

Intelligent  agent  technology  holds  much  promise  for  on-line  and  off-line  performance  feedback 
and  tutoring  in  complex  environments,  and  many  efforts  are  now  being  reported  (Tecuci  &  Keeling, 
1998;  Towns,  Fitzgerald,  &  1998,  to  name  just  a  few).  Flowever,  existing  difficulties  in  modeling  expert 
performance  in  these  settings  leads  one  to  the  conclusion  made  by  Salas,  Cannon-Bowers,  and  their 
colleagues-pertaining  to  the  importance  of  event-driven  assessment  of  performance  (Fowlkes,  Lane, 
Salas,  Franz,  &  Oser,  1994;  Salas  et  al.,  1997a,b,c).  It  is  absolutely  critical  to  develop  scenarios  with 
careful  attention  to  research  and  training  goals,  individual  and  team-level  constructs  of  performance,  with 
well-defined  manipulations,  articulation  of  expected  effects,  and  interpretations  of  behavioral  responses 
to  trigger  events. 

In  summary,  we  have  described  progress  to  date  in  our  investigations  in  team  training  and 
research.  Further  efforts  will  continue  along  the  path  described  here.  Infrastructure  for  team  training 
research  has  been  established,  with  regard  to  software,  hardware,  and  facilities  (Regian  &  Elliott,  2000). 
Propositions  based  on  instructional  principles  and  boundary  conditions  will  be  further  explicated  and 
tested.  1CA1  principles  and  instructional  approach  will  be  applied  to  team  settings  and  embedded  in 
team-based  research  systems.  These  efforts  will  not  be  achieved  by  a  single  researcher  or  research  lab. 
Instead,  we  will  collaborate  with  a  number  of  universities  and  companies,  as  part  of  a  multidiscipline 
team  of  teams,  focused  on  the  enhancement  of  computer-based  training  and  performance  in  complex 
naturalistic  settings. 
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